This document discusses using Hadoop MapReduce on Amazon Web Services for parallel processing of large datasets in a distributed manner. It provides an example of analyzing Wikipedia page view logs stored in S3 using Hadoop Streaming to run MapReduce jobs on EC2 instances with mappers and reducers written in Perl scripts. The jobs count page views to different pages and output the results.

1. QUICK AND DIRTY PARALLEL PROCESSING ON THE CLOUD Daniel Sikar

2. EC2 S3

5. Elastic MapReduce Ruby library

6. Hadoop

7. s3cmd

8. Hadoop MapReduce Job Tracker + Task Tracker + Slaves HDFS – Distributed file system

9. Hadoop MapReduce usage Data crunching in general Clicks Statistics etc

10. Hadoop Project Mgmt Committee

11. MapReduce ?

12. MapReduce Key Pairs <key,value>

13. MapReduce

14. HTTP Logs Log file A: (...) FreeTouchScreenNokia5230 (...) (...) GetRidofAllSpeedCameras(...) (...) USManWinsLottery (...) (...) BNPToLaunchElectionManifesto (...) Log file B: (...) FreeTouchScreenNokia5230 (...) (...) BodyLanguageTellsAll (...)

15. MapReduce <FreeTouchScreenNokia5230, 1> + <FreeTouchScreenNokia5230, 1> = <FreeTouchScreenNokia5230, 2>

16. Hadoop Streaming Running MapReduce jobs with .exe fiels and scripts $ <list> | mapper | reducer

17. Hadoop Streaming Running MapReduce jobs with .exe fiels and scripts $ <list> | mapper | reducer

18. Real life example of Hadoop Streaming usage

19. Wikipedia Page Access Logs

20. Wine Grape Varieties

21. Wikipedia WGV Page Access Stats

22. Business Decisions

23. Launching a virtual Hadoop Cluster $ elastic-mapreduce --create --name &quot;Wiki log crunch&quot; --alive --num-instances –instance-type c1.medium 20 Created job flow <job flow id> $ ec2din (...)

26. Pseudo-Distributed Operation

27. Fully-Distributed Operation

28. NameNode

29. JobTracker

30. DataNode + TaskTracker

32. Pseudo-Distributed Operation

33. Fully-Distributed Operation

34. NameNode

35. JobTracker

36. DataNode + TaskTracker

37. Add a step $ elastic-mapreduce --jobflow <jfid> --stream --step-name &quot;Wiki log crunch&quot; --input s3n://dsikar-wikilogs-2009/dec/ --output s3n://dsikar-wikilogs-output/21 --mapper s3n://dsikar-wiki-scripts/wikidictionarymap.pl --reducer s3n://dsikar-wiki-scripts/wikireduce.pl http://<instance public dns>:9100

38. s3cmd # make bucket $ s3cmd mb s3://dsikar-wikilogs # put log files $ s3cmd put pagecounts-200912*.gz s3://dsikar-wikilogs/dec $ s3cmd put pagecounts-201004*.gz s3://dsikar-wikilogs/apr # list log files $ s3cmd ls s3://dsikar-wikilogs/ # put scripts $ s3cmd put *.pl s3://dsikar-wiki-scripts/ # delete log files $ s3cmd del --recursive --force s3://dsikar-wikilogs/ # remove bucket $ s3cmd rb s3://dsikar-wikilogs/

39. Elastic MapReduce --create --list --jobflow --describe --stream --terminate

40. Output files part-00000 part-00001 part-00002 (...)

41. Further aggregation

42. Conclusion Hadoop MapReduce provides out-of-the-box ready-to-go distributed computing.